The present disclosure relates to an imaging device, an image processing method, and a program. More specifically, the present disclosure relates to an imaging device, an image processing method, and a program that perform correction of image degradation, for example, due to blooming.
In recent years, a CCD image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor has been used as a solid-state image sensor such as a video camera or a digital still camera. In such an image sensor, the number of pixels is rapidly increasing or a size of the image sensor is being rapidly reduced.
However, with the increase in the number of pixels or the size reduction, an interval between adjacent pixels in the image sensor becomes small, and a new problem of charge leakage between the pixels, which is called blooming, is generated.
Blooming refers to a phenomenon in which one of adjacent pixels is in a saturation state and charges overflow and leak to the other pixel.
The blooming phenomenon will be described with reference to FIG. 1.
When strong light is incident on one of adjacent pixels, weak light is incident to the other pixel, and a light receiving part 1 on which the strong light is incident is in a saturation state, charges generated by the light incident on the light receiving part 1 are not accumulated in the light receiving part 1 and some of the charges leak to an adjacent light receiving part 2. This phenomenon is called blooming.
Here, to classify the pixels, the pixel that generates charge overflowing, that is, the blooming generation pixel, is defined as an aggressor pixel and the pixel that accumulates charges overflowing from the aggressor pixel, that is, the blooming receiving pixel, is defined as a victim pixel.
When such blooming is generated, a pixel value of the pixel that generates charge overflowing, that is, the aggressor pixel that generates the blooming, becomes a saturation value. Meanwhile, a pixel value of the pixel accumulating the overflowing charges, that is, the victim pixel that receives the blooming, is set to an incorrect pixel value due to the overflowing charges.
FIG. 2 is a diagram illustrating transitions of accumulated charges of an aggressor pixel and a victim pixel due to blooming.
Temporal changes of charge amounts when light is incident on the aggressor pixel and the victim pixel adjacent to each other are shown. At a time t1, the aggressor pixel is saturated and charge overflowing occurs.
The overflowing charges flow into the adjacent victim pixel and an accumulated charge amount of the victim pixel is changed.
A dotted line corresponding to the victim pixel shown in FIG. 2 indicates a transition of normal accumulated charges when there is no blooming, and a solid line indicates a transition of the accumulated charges changed due to blooming.
Adjacent pixels have different sensitivity according to a wavelength of incident light and a characteristic of a color filter, but when a pixel having high sensitivity (an aggressor) is saturated, a pixel having low sensitivity (a victim) is affected.
When the incident light is constant, a change in a charge amount will have linearity. However, the victim pixel has nonlinearity due to the influence of blooming As a result, there is a problem in that an image is discolored.
In general, since the influence of blooming is generated in the vicinity of saturation of an image, a technique for eliminating the influence of blooming through a solution process of thinning a color component in the vicinity of the saturation is often used.
An example of this solution to blooming is disclosed in, for example, Japanese Patent Application Laid-Open Publication No. 2008-294698.
Japanese Patent Application Laid-Open Publication No. 2008-294698 describes a blooming solution for a captured image of a wide dynamic range image.
A wide dynamic range image will be simply described. Image capture environments by an imaging device are assumed to be various environments: indoors or outdoors and daytime or nighttime. When the image sensor (an image sensor) is used in the various environments, it is necessary for sensitivity to have an optimal value by controlling a charge accumulation time of a photoelectric conversion element, that is, adjusting an exposure time according to brightness of a photography environment.
For example, an effective pixel value can be output from each pixel by suppressing occurrence of saturation of the pixel value or sensitivity insufficiency by setting a short exposure time in a bright environment and a long exposure time in a dark environment.
However, for example, when an image in which both a bright subject and a dark subject are present is captured, early charge accumulation begins to become saturated in a photographed pixel of the bright subject if photography is performed in a constant exposure time. However, an imbalance in which sufficient charge accumulation is not achieved may be generated in a photographed pixel of the dark subject.
A technique of outputting an effective pixel value from the bright subject to the dark subject is a process of extending a dynamic range, and such an imaging process is called high dynamic range (HDR) imaging or wide dynamic range (WDR) imaging.
Hereinafter, an image capture technique for outputting an effective pixel value from the bright subject to the dark subject will be described as an HDR image capture technique.
The following two techniques have been proposed as HDR image capture techniques.
(Technique 1) Multi-shot technique: a technique of generating a wide dynamic range image through a process of combining a long-time exposure image and a short-time exposure image using a plurality of images continuously captured in a plurality of different exposure times.
(Technique 2) One-shot technique: a technique of generating a wide dynamic range image by combining a long-time exposure pixel and a short-time exposure pixel included in one image by setting, for example, different exposure times in units of rows of a pixel arrangement without continuously capturing a plurality of images.
The multi-shot technique is disclosed in, for example, Japanese Patent Application Laid-Open Publication Nos. Hei 2-174470, Hei 7-95481, and Hei 11-75118.
Specifically, as shown in FIG. 3, a short-time exposure image (low-sensitivity image) whose exposure time is short, and a long-time exposure image (high-sensitivity image) whose exposure time is long are alternately captured for each frame.
These captured images having the different exposure times are accumulated in a frame memory, and signal processing such as selection and acquisition of an effective pixel value from each image is executed. For example, an image having a high dynamic range is generated through signal processing, such as preferential acquisition of a pixel value of a short-time exposure image (low-sensitivity image) for a bright subject area and preferential acquisition of a pixel value of a long-time exposure image (high-sensitivity image) for a dark subject area.
In Japanese Patent Application Laid-Open Publication No. 2008-294698 described above, a blooming correction process for an HDR image by such a multi-shot technique has been described.
(a) A short-time exposure image, and
(b) a long-time exposure image,
which are continuously captured by a multi-shot technique are shown in FIG. 4.
Japanese Patent Application Laid-Open Publication No. 2008-294698 describes blooming correction when an exposure time of one entire image is the same, that is, all exposure times of adjacent pixels are the same, as shown in FIG. 4.
A configuration in which an amount of blooming generated between adjacent pixels of a long-time exposure pixel is estimated using a pixel value of a short-time exposure image, and correction of a pixel value of a long-time exposure image is performed based on this estimation is disclosed.
Even when blooming is generated in the long-time exposure image, a continuously captured short-time exposure image does not have nonlinearity such as saturation or blooming since the exposure time is sufficiently short in the continuously captured short-time exposure image.
Due to this characteristic, in the case of an HDR captured image by multi-shot, an amount of blooming generated between the adjacent pixels of the long-time exposure image is estimated based on a pixel value of the short-time exposure image, and correction of the long-time exposure pixel can be performed based on the estimation value.
Japanese Patent Application Laid-Open Publication No. 2008-294698 describes a blooming correction process for the HDR image by such a multi-shot technique.
However, it is difficult for a blooming correction process disclosed in Japanese Patent Application Laid-Open Publication No. 2008-294698 to apply to an image captured by a one-shot technique (Technique 2) of the HDR image capture technique described above.
The one-shot technique (Technique 2) is a technique of generating a wide dynamic range image by combining a long-time exposure pixel and a short-time exposure pixel included in one image by setting, for example, different exposure times in units of rows of a pixel arrangement without continuously capturing a plurality of images.
Further, a photography process of an HD image using a one-shot technique is disclosed in, for example, Japanese Patent Application Laid-Open Publication No. 2006-253876 and Japanese Patent Application Publication No. 2006-542337.
For a captured image using the one-shot technique, both a long-time exposure pixel and a short-time exposure pixel are present in one image.
Accordingly, it is difficult to perform a process using the two images as shown in FIG. 4:
(a) short-time exposure image, and
(b) long-time exposure image.
For example, one captured image is an image in which both low-sensitivity pixels that are short-time exposure pixels and high-sensitivity pixels that are long-time exposure pixels are present, as shown in FIG. 5.
Thus, a plurality of pixels with sensitivity are present in the image captured by the one-shot technique. In such an image, blooming is generated in a very complex form.
That is, the following blooming between various pixels is generated in the image:                Blooming from a long-time exposure pixel to a long-time exposure pixel        Blooming from a short-time exposure pixel to a short-time exposure pixel        Blooming from a long-time exposure pixel to a short-time exposure pixel        Blooming from a short-time exposure pixel to a long-time exposure pixelAccordingly, it is necessary to correct blooming in consideration of such blooming generation forms.        
In Japanese Patent Application Laid-Open Publication No. 2008-294698 described above, a blooming correction amount for an HDR captured image by a multi-shot technique has been estimated using a short-time exposure image not affected by blooming.
However, in the case of an HDR image captured in one shot, since blooming correction must be performed using a short-time exposure image not affected by blooming, for example, a process for reducing an error of the correction amount is necessary.
Further, Japanese Patent Application Laid-Open Publication No. 2008-147818 discloses a method of minimizing blooming generation by changing reading timing control for an image sensor rather than by performing signal processing.
In this method, since a reading timing is discontinuous according to positions of the image, for example, a frame memory is necessary to obtain continuous data. Further, when the exposure time is desired to be changed, a data reading timing is changed according to an exposure time and control for signal processing in a subsequent stage is difficult.